METHOD FOR SCREEN EDGE DETECTION

专利摘要:
A method of detecting labels comprises: acquiring, by an image processing control, an image representing a shelf; increasing an intensity of a foreground subset of image pixels exceeding an upper intensity threshold, and decreasing an intensity of a background subset of pixels below a lower intensity threshold; in response to increasing and decreasing, (i) determining gradients for each of the pixels and (ii) selecting a candidate set of pixels based on the gradients; lay a plurality of shelf message lines over the image derived from a candidate set of pixels throughout; identify a pair of the shelf candidate lines that satisfy a predetermined shelf edge boundary frame corresponding to the pair of shelf candidate lines.
公开号:BE1025892B1
申请号:E2018/5620
申请日:2018-09-07
公开日:2019-09-12
发明作者:Richard Jeffrey Rzeszutek；Vlad Gorodetsky
申请人:Symbol Technologies Llc；
IPC主号:

专利说明:

METHOD FOR SCREEN EDGE DETECTION
Background
Environments in which inventory inventory is managed, such as products for sale in a retail environment, can be complex and fluid. For example, a particular environment can include a wide variety of objects with different dimensions, shapes, and other properties. Such objects can be supported by shelves in a variety of positions and orientations. The variable position and orientation of the objects, as well as the variation in exposure and the placement of labels and other indicators on the objects and the shelves, can make detection of structural properties, such as the edges of the shelves, more difficult. Summary of the invention
According to the invention, a method is provided for detecting a shelf edge by an image processing controller, comprising obtaining, by the image processing controller, an image representing a shelf, increasing an intensity of a foreground subset of image pixels in the image, wherein each pixel in the foreground subset of pixels has an initial intensity that exceeds an upper intensity threshold, and lowering an intensity of a background subset of pixels in the image, wherein each pixel in the background subset of pixels has an initial intensity lower than a lower intensity threshold in response to the increasing and decreasing, (i) determining gradients for each of the pixels and (ii) selecting a candidate set of the pixels based on the gradients, superimposing a plurality of shelf candidate lines over the image derived from the candidate set of pixels, identifying a few of the shelf candidate lines which satisfy a predetermined sequence of intensity transitions, and
BE2018 / 5620 generating a shelf edge boundary frame corresponding to the pair of shelf candidate lines.
The method may further include downsampling the image prior to raising and lowering.
The method may further comprise adjusting an intensity of an intermediate subset of pixels with intensities between the upper intensity threshold and the lower intensity threshold.
The method may preferably further include, prior to increasing, decreasing, and adjusting, intensity adjustment parameters by comparing pixel populations of the foreground subset and the background subset with foreground and background target populations.
Selecting the candidate set of pixels may preferably include selecting pixels which locally exhibit maximum gradients in an orientation perpendicular to an expected orientation of the shelf edge.
Overlapping the plurality of shelf candidate lines may preferably assign each pixel in the candidate set to one of a plurality of bins, each bin corresponding to a line, select candidate subsets of the pixels assigned to respective bins with pixel populations that exceed a population threshold value, and generate of a shelf candidate line of each candidate subset of pixels.
Identifying can further include traversing the image in a predetermined direction, and evaluating successive pairs of the shelf candidate lines against the intensity sequence.
The identification may further determine whether the pair of shelf candidate lines that meets the predetermined sequence meets a distance criterion.
BE2018 / 5620
The distance criterion can define a minimum distance between the pair of shelf candidate lines and a maximum distance between the pair of shelf candidate lines.
The method may further include storing the shelf edge limiting frame in a memory.
According to another aspect of the invention, a computer device is provided for detecting shelf edges, comprising a memory; and an image processing control coupled to the memory, the image processing control comprising: a pre-processor arranged to obtain an image representing a shelf, and to increase an intensity of a foreground subset of pixels in the image, wherein each pixel in the foreground subset of pixels has initial intensity that exceeds an upper intensity threshold, and decreases an intensity of a background subset of pixels in the image, wherein each pixel in the background subset of pixels has an initial intensity that is lower than a lower intensity threshold; a selector arranged to, in response to the raising and lowering, (i) determine gradients for each of the pixels and (ii) select a candidate set of the pixels based on the gradients; a line generator adapted to superimpose a plurality of shelf candidate lines over the image derived from the candidate set of pixels; and a region generator adapted to identify a pair of the shelf candidate lines that satisfy a predetermined sequence of intensity transitions; and the image processing controller adapted to generate a shelf edge boundary frame corresponding to the pair of shelf candidate lines in the memory.
The pre-processor may further be arranged to downsample the image prior to raising and lowering.
BE2018 / 5620
The pre-processor may further be adapted to adjust an intensity of an intermediate subset of pixels with intensities between the upper intensity threshold and the lower intensity threshold.
The processor may further be arranged to determine, prior to raising, decreasing, and adjusting, intensity adjustment parameters by comparing pixel populations of the foreground subset and the background subset with foreground and background target populations.
The selector may further be arranged to select the candidate set of pixels by selecting pixels which locally display maximum gradients in an orientation perpendicular to an expected orientation of the shelf edge.
The line generator may preferably be arranged to overlap the plurality of shelf candidate lines by assigning each pixel in the candidate set to one of a plurality of bins each corresponding to a line, selecting candidate subsets of the pixels assigned to respective bins with pixel populations which have a threshold value exceed, and generate a shelf candidate line from each candidate subset of pixels.
The region generator may be adapted to identify the pair of shelf candidate lines by traversing the image in a predetermined direction, and evaluate successive pairs of the shelf candidate lines against the intensity sequence.
The region generator can further be arranged to identify the pair of shelf candidate lines by determining whether the pair of shelf candidate lines that meets the predetermined sequence meets a distance criterion.
The distance criterion can, for example, define a minimum distance between the pair of shelf candidate lines and a maximum distance between the pair of shelf candidate lines.
BE2018 / 5620
The image processing control can further be arranged to store the shelf edge limiting frame in the memory.
The present invention will be further elucidated with reference to figures of exemplary embodiments. Corresponding elements are indicated with corresponding reference numerals.
Brief description of the various views of the drawings
The accompanying figures, in which the same reference numerals refer to identical or functionally identical elements throughout all individual views, are included in, and form part of the description, together with the detailed description below, and serve to further clarify embodiments of concepts which represent the claimed invention. and clarify various principles and advantages of these embodiments.
FIG. 1 is a schematic diagram of a mobile automation system.
FIG. 2A shows a mobile automation device in the system of FIG. 1.
FIG. 2B is a block diagram of certain internal hardware components of the mobile automation device in the system of FIG. 1.
FIG. 2C is a block diagram of certain internal hardware components of the server in the system of FIG. 1.
FIG. 3 is a flow chart of a method for shelf detection.
FIG. 4 is an example of a shelf image obtained in the implementation of the method according to FIG. 3.
FIG. 5A is an example of a sigma-shaped function curve used to adjust pixel intensities in the image or FIG. 4.
BE2018 / 5620
FIG. 5B is a modified version of the image of FIG. 4, obtained through application of the adjustment function of FIG. 5A.
FIG. 6 is a method for determining adjustment parameters for generating the image of FIG. 5B.
FIG. 7A is a method for generating shelf candidate lines in the application of the method of FIG. 3.
FIG. 7B is a graphical representation of a modified Heat transformation used in the application of the method of FIG. 7A.
FIG. 8 shows the image of FIG. 5B with the output of the method of FIG. 7A.
FIG. 9 is a method for identifying candidate line pairs in the method of FIG. 3.
FIG. 10 is a method for generating boundary frames based on the lines generated via the method of FIG. 7A.
FIGS. 11-12 show the results of the method of FIG. 10.
Those skilled in the art will appreciate that parts are shown in the figures for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions and positions of some parts in the figures may be exaggerated relative to other parts, to help increase understanding of the embodiments of the present invention.
The apparatus and method components, when appropriate, are represented in the figures by conventional symbols, which show only those specific details relating to the understanding of the embodiments of the present invention, so as not to obscure the description with details that are easy to understand. be clear to the person skilled in the art who has the advantage of the present description.
BE2018 / 5620
Detailed description
In retailing environments in which a plurality of products are carried on shelves, mobile automation systems may be arranged to take images of the shelves and to determine, from the images, various information regarding the products. For example, price labels can be located and decoded within the image, for use in ensuring that products are labeled with the correct prices. Furthermore, gaps between products on the shelves can be identified as potentially indicating that one or more products are out of stock and require refilling. The above provisions may require the identification of shelf edges (i.e., the substantially vertical surfaces where the substantially horizontal bearing surfaces of the shelves cease; the shelf edges are directed to aisles in the retail environment) in the images for use as reference structures for identification of labels, products, holes, and the like. In addition, in certain embodiments, a mobile automation device relies on identification of shelf edges to assist with its navigation within the retail environment.
The identification of shelf edges from image data is complex due to a variety of factors, including but not limited to the proximity of products with a wide variety of shapes, colors and orientations to the shelf edges. Such factors also include variations in lighting, reflections, obstructions of products or other objects, and the like. Furthermore, the shelf edges can have various other elements, such as product labels, provided thereon, which changes the appearance of parts of the shelf edges into images.
Examples discussed herein are directed to a method of detecting a shelf edge by an image processing controller, comprising: obtaining, by
BE2018 / 5620 image processing control, of an image representing a shelf; increasing an intensity of a foreground subset of image pixels in the image, wherein each pixel in the foreground subset of pixels has an initial intensity that exceeds an upper intensity threshold, and decreasing an intensity of a background subset of pixels in the image, wherein each pixel in the image background subset of pixels has an initial intensity that is lower than a lower intensity threshold; in response to increasing and decreasing, (i) determining gradients for each of the pixels and (ii) selecting a candidate set of the pixels based on the gradients; superimpose a plurality of shelf candidate lines over the image derived from the candidate set of pixels; identify a pair of shelf candidate lines that satisfy a predetermined sequence of intensity transitions; and generating a shelf edge boundary frame corresponding to the pair of shelf candidate lines.
Further examples shown herein are directed to a computer apparatus for detecting shelf edges, comprising: a memory; and an image processing control coupled to the memory, the image processing control comprising: a pre-processor arranged to obtain an image representing a shelf, and to increase an intensity of a foreground subset of pixels in the image, wherein each pixel in the foreground subset of pixels has initial intensity that exceeds an upper intensity threshold, and decreases an intensity of a background subset of pixels in the image, wherein each pixel in the background subset of pixels has an initial intensity that is lower than a lower intensity threshold; a selector arranged to, in response to the raising and lowering, (i) determine gradients for each of the pixels and (ii) select a candidate set of the pixels based on the gradients; a line generator adapted to superimpose a plurality of shelf candidate lines over the image derived from the candidate set of pixels; and a region generator adapted to identify a pair of the
BE2018 / 5620 shelf candidate lines that satisfy a predetermined sequence of intensity transitions; and the image processing controller adapted to generate a shelf edge boundary frame corresponding to the pair of shelf candidate lines in the memory.
FIG. 1 shows a mobile automation system 100 according to the teachings of this description. The system 100 includes a server 101 communicatively connected to at least one mobile automation device 103 (also simply referred to herein as the device 103) and at least one client computer device 105 through communication links 107, illustrated in the present example as including wireless connections. In the current example, the connections 107 are provided by a Wireless Local Area Network (WLAN) deployed within the retail environment through one or more access points. In other examples, the server 101, the client device 105, or both, are provided outside the retail environment, and the connections 107 therefore include wide-area networks such as the internet, mobile networks, and the like. As will be described in more detail below, the system 100 also includes a charging station 108 for the device 103. The station 108 is communicatively connected to the server 101 via a connection 109 which in the present example is a wired connection (e.g., an Ethernet -link). However, in other examples, the connection 109 is a wireless connection.
The client computer device 105 is shown in FIG. 1 shown as a mobile computer device, such as a tablet, smartphone or the like. In other examples, the client device 105 includes computer devices such as a desktop computer, a laptop computer, another server, a kiosk, a display, or other suitable device. The system 100 can include a plurality of client devices 105, each communicating with the server 101 via respective connections 107.
BE2018 / 5620
In the example shown, the system 100 is deployed in a retail environment comprising a plurality of shelf modules 110-1, 110-2, 110-3 and so on (collectively called shelves 110, and generically called a shelf 110 - this naming is also used for other elements as discussed here). Each shelf module 110 carries a plurality of products 112. Each shelf module 110 includes a shelf ridge 116-1, 116-2, 116-3 and a support surface (e.g., support surface 117-3 as shown in FIG. 1) extending from the shelf ridge 116 to a shelf edge 118-1, 118-2, 118-3. The shelf modules 110 are typically arranged as a plurality of aisles, each aisle comprising a plurality of modules that are aligned end-to-end. In such embodiments, the shelf edges 118 are oriented into the aisles, aisles through which customers can move in the retail environment as well as the device 103.
More specifically, the device 103 is deployed within the retail environment, and communicates with the server 101 (via the connection 107) to navigate along the length 119 of at least a portion of the shelves 110, autonomously or partially autonomously. The device 103 is equipped with a plurality of navigation and data acquisition sensors 104, such as image sensors (for example, one or more digital cameras) and depth sensors (for example, one or more Light Detection and Ranging (LIDAR) sensors), and is further adapted to accommodate the sensors in to obtain shelf data. In the present example, the device 103 is arranged to obtain a series of digital images from the shelves 110, as well as a series of depth measurements, each of which is the distance and direction between the device 103 and one or more points on a shelf 110, such as the shelf itself or describe the product placed on the shelf.
The server 101 includes an image processing controller with a special purpose, such as a processor 120, specifically designed to control the mobile automation device 103 to obtain data (e.g., images from the shelves 110), to obtain the recorded data via a
BE2018 / 5620 communication interface 124 and storing the recorded data in a storage location 132 in a memory 122. The server 101 is further arranged to perform various post-processing operations on the recorded data and to detect certain structural features - such as the shelf edges 118 - within the recorded data. The post-processing of recorded data by the server 101 will be discussed in more detail below. The server 101 can also be arranged to determine product status data based in part on the aforementioned shelf detections, and to send status notifications (e.g. notifications indicating that products are out of stock, low in stock or out of place) to the mobile device 105 as response to determining product status data.
The processor 120 is interconnected with a non-perishable computer-readable storage medium, such as the above-mentioned memory 122, with thereon computer-readable instructions for performing control of the device 103 to obtain data, as well as the aforementioned post-processing functionality, as discussed in more detail below. The memory 122 comprises a combination of volatile (e.g. Random Access Memory or RAM) and non-volatile memory (e.g. Read Only Memory or ROM, Electrically Erasable Programmable Read Only Memory or EEPROM, flash memory). The processor 120 and the memory 122 each comprise one or more integrated circuits. In one embodiment, the processor 120 further comprises one or more central processing units (CPUs) and / or graphic processing units (GPUs). In one embodiment, a specially designed integrated circuit, such as a Field Programmable Gate Array (FPGA), is designed to perform the shelf detection as discussed herein, either as an alternative or as an addition to the image processing controller / processor 120 and memory 122. As one skilled in the art It will be appreciated that the mobile automation device 103 includes
BE2018 / 5620 also includes one or more controllers or processors and / or FPGAs, communicatively connected to the controller 120, specifically arranged to control navigation and / or data acquisition aspects of the device 103. The client device 105 also includes one or more controllers or processors and / or FPGAs, communicating with the controller 120, specifically arranged to process (e.g. display) notifications received from the server 101.
The server 101 also includes the aforementioned communication interface 124 interconnected with the processor 120. The communication interface 124 includes appropriate hardware (e.g., transmitters, receivers, network interface controllers, and the like) that enable the server 101 to communicate with other Computer devices - in particular the device 103, the client device 105 and the station 108 - via the connections 107 and 109. The connections 107 and 109 can be direct connections, or connections which traverse one or more networks, including both local and wide-area networks. The specific components of the communication interface 124 are selected based on the type of network or other connections over which the server 101 is required to communicate. In the current example, as noted earlier, a wireless local area network has been implemented within the retail environment by deploying one or more wireless access points. The connections 107 therefore include a wireless connection between the device 103 and the mobile device 105 and the access points mentioned above and / or a wired connection (e.g. an Ethernet-based connection) between the server 101 and the access point.
The memory 122 stores a plurality of applications, each comprising a plurality of computer-readable instructions executable by the processor 120. The execution of the above-mentioned instructions by the processor 120 sets the server 101 to
BE2018 / 5620 various actions as discussed herein. The applications stored in the memory 122 include a control application 128, which may also be implemented as a series of logically separate applications. Generally, through execution of the control application 128 or sub-components thereof, the processor 120 is arranged to implement various functionality. The processor 120, configured in the execution of the control application 128, is also referred to herein as the controller 120. As will now become apparent, some or all of the functionality implemented by the controller 120 as described below can also be performed by pre-configured hardware elements (e.g. one or more Application-Specific Integrated Circuits (ASICs) instead of execution of the control application 128 by the processor 120.
Now looking at FIGS. 2A and 2B, the mobile automation device 103 is shown in more detail. The device 103 comprises a chassis 201 comprising a movement mechanism 203 (for example, one or more electric motors which drive wheels, tracks, or the like). The device 103 further includes a sensor mast 205 carried by the chassis 201 and which, in the present example, extends upwardly (e.g., substantially vertically) from the chassis 201. The mast 205 carries the sensors 104 previously mentioned. In particular, the sensors 104 include at least one image sensor 207, such as a digital camera, as well as at least one depth sensing sensor 209, such as a 3D digital camera. The device 103 also includes additional depth sensors, such as LIDAR sensors 211. In other examples, the device 103 includes additional sensors, such as one or more RFID readers, temperature sensors, and the like.
In the current example, the mast 205 carries seven digital cameras 207-1 to 207-7, and two LIDAR sensors 211-1 and 211-2. The mast 205 also carries a plurality of exposure devices 213 arranged to
BE2018 / 5620 to illuminate the fields of vision of the respective cameras 207. That is, the exposure device 213-1 illuminates the field of view of the camera 207-1, and so on. The sensors 207 and 211 are oriented to the mast 205 such that the fields of view of each sensor face a shelf 110 along the length 119 along which the device 103 is moving. The device 103 is arranged to follow a location of the device 103 (e.g., a location of the center of the chassis 201) in a shared reference frame that was previously located in the retail facility, which allows data to be recorded by the mobile automation device to be registered on the shared frame of reference.
To that end, the mobile automation device 103 includes a controller with a special purpose, such as a processor 220, as shown in FIG. 2B. Interconnected with a non-perishable computer-readable storage medium, such as a memory 222. The memory 222 comprises a combination of volatile (e.g. Random Access Memory or RAM) and non-volatile memory (e.g. Read Only Memory or ROM, Electrically Erasable Programmable Read Only Memory or EEPROM, flash memory). The processor 220 and the memory 222 each comprise one or more integrated circuits. The memory 222 stores computer-readable instructions for execution by the processor 220. In particular, the memory 222 stores a control application 228 which, when executed by the processor 220, configures the processor 220 to perform various functions related to navigation from the device 103 (e.g. by controlling the moving mechanism 203) and to detecting shelf edges in data obtained by the sensors (e.g., the cameras 207). The application 208 can also be implemented as a set of separate applications in other examples.
The processor 220, when configured so by the execution of the application 228, can also be referred to as one
BE2018 / 5620 controller 220, or in the context of shelf edge detection from obtained data, as image processing controller 220. Those skilled in the art will understand that the functionality implemented by the processor 220 through the execution of the application 228 may also be implemented by one or more specially designed hardware and firmware components such as FPGAs, ASICs and the like in other embodiments.
The memory can also store a storage location 232 comprising, for example, a map of the environment in which the device 103 operates, for example during the execution of the application 228. The device 103 can communicate with the server 101, for example to receive instructions to initiate data acquisition operations, via a communication interface 224 via the connection 107 shown in FIG. 1. The communication interface 224 also allows the device 103 to communicate with the server 101 through the station 108 and the connection 109.
In the present example, as discussed below, one or both of the server 101 (as set via the execution of the control application 128 by the processor 120) and the mobile automation device 103 (as set via the execution of the application 228 by the processor 220) arranged to process image data captured by the mobile automation device 103 to identify portions of the captured data representing the shelf edges 118. In further examples, the image data processing as discussed below may be performed on a computer device other than the server 101 and the mobile automation device 103, such as the client device 105. The image data processing as mentioned above will be described in more detail in relation to its operation on the server 101, via implementation of the application 128.
Now looking at FIG. 2C, for describing the operation of the application 128 to identify the shelf edges 118 from acquired image data, certain parts of the application 128 will be more
BE2018 / 5620 detail. As will be apparent to those skilled in the art, in other examples, the parts of the application 128 may be separated into different distinct applications, or combined into other sets of parts. Some or all of the components illustrated in FIG. 2C can also be implemented as special hardware components, such as one or more ASICs or FPGAs. For example, in one embodiment, to improve reliability and processing speed, at least some of the components of FIG. 2C can be programmed directly into the image processing controller 120, which can be an FPGA or an ASIC with circuit and memory configuration specifically designed to process image processing of a large volume of sensor data received from the mobile automation device 103. In such an embodiment, part or all of the control application 128 discussed below is an FPGA or an ASIC chip.
The control application 128 comprises an image pre-processor 200 adapted to obtain an image representing a portion of the shelves 110 and the products 112 carried thereon and adapted to pre-process the image, for example by adjusting the contrast of the image. The control application 128 also includes a selector 204 adapted to select a candidate set of pixels from the pre-processed image (i.e., the output from the pre-processor 200); as will be discussed below, the candidate set of pixels are pixels in the image that have an increased chance of displaying shelf edges 118. The control application 128 also includes a line generator 208 adapted to generate and superimpose shelf candidate lines derived from the candidate set of pixels. The shelf candidate lines, as will be seen below, represent potential locations within the pre-processed image which shelf edges 118 represent. Furthermore, the control application 128 includes a region generator 212 adapted to use regions of the shelf candidate lines to define regions
BE2018 / 5620 identify the pre-processed image which is likely to display the shelf edges 118.
The functionality of the control application 128 will now be described in more detail. Looking at FIG. 3, a method 300 of shelf detection is shown. The method 300 will be described in conjunction with its performance on the system 100 and with reference to the components illustrated in FIG. 2C. However, as previously noted, in other examples, part or all of the method 300 is performed by the parts illustrated in FIG. 2B.
At block 305, the controller 120, and in particular the pre-processor 200 of the control application 128, is adapted to obtain a digital image of one or more shelf modules 110 and the products 112 carried thereon. The image obtained at block 305 is, for example, recorded by the device 103 and stored in the storage location 132. The pre-processor 200 is therefore arranged, in the above example, to obtain the image by retrieving the image from the storage location 132.
An example image 400 is shown in FIG. 4, which represents parts of two shelf edges 118-3a and 118-3b, as well as products 112 carried by the bearing surfaces (not visible in FIG. 4) corresponding to each shelf edge 118. As previously noted, each shelf edge 118 in the image 400 is represented as an elongated rectangular substantially vertical surface directed to an aisle in which the shelf is located (i.e., directed to the device 103 which moves through the aisle to obtain the image 400). A shelf back 416 is also visible in the image 400.
At block 307, after having acquired the image 400, the pre-processor 200 may also be arranged to downsample the image 400 before further processing is performed to detect the shelf edges 118 in the image 400. For example, the pre-processor may be 200
BE2018 / 5620 arranged at block 307 to generate a downsampled version of the image 400 with a resolution of about one-tenth of the original resolution of the image 400 (e.g., a downsampled version, a resolution of about 0.5 megapixels can be generated from an original image with a resolution of approximately 5 megapixels). Other useful orders of downsampling can also be implemented depending on the resolution of the original image. In the discussion below, the processing functions performed by the control application 128 are referred to as being executed on the image 400. It is assumed that when block 307 is executed, those processing functions are performed on the downsampled version of the image 400. When the block 307 is omitted, those processing functions are instead performed on the original image 400.
From block 305 or block 307, the execution of the method 300 proceeds to block 310, where the control application 128, and in particular the pre-processor 200, is adapted to adjust the contrast of the image 400. As will become apparent, each pixel in the image 400 has an intensity level. The nature of the intensity level depends on the data format used by the image 400. For example, the intensity level can be expressed as a value between zero, corresponding to black, and 255, corresponding to white. In other examples, the intensity value can be expressed as a value between 0, corresponding to black, and 1, corresponding to white. In further examples, the intensity value can be expressed as a value between 0 (black) and 65535 (white). The pre-processor 200 is arranged to identify at least two pixel subsets in the image 400: a foreground subset, the pixels of which each have an original intensity that exceeds an upper intensity threshold; and a background subset, the pixels of which each have an original intensity that falls below a lower intensity threshold. The pre-processor is 200
BE2018 / 5620 adapted to, after having identified the subsets mentioned above, to increase the intensity of the foreground pixels, and to decrease the intensity of the background pixels. In the current example, the intensities of the foreground subset of pixels are increased to a common upper level; more specifically, the intensities of the foreground subset of pixels are increased to a maximum level (e.g., 255, or white). On the other hand, the intensities of the background subset are lowered to a common lower level, which in the current example is a minimum level (e.g., 0, or black).
The identification of foreground and background pixel subsets can be performed by comparing the intensity of each pixel of the image 400 with a predetermined upper intensity threshold and a predetermined lower intensity threshold. In other examples, the upper and lower intensity thresholds are dynamically determined for each image processed according to the method 300 (i.e., for each embodiment of the method 300). Further, in addition to the foreground and background pixel subsets, the pre-processor 200 is arranged to identify an intermediate subset of pixels in some examples (e.g., all pixels not identified as members of either the foreground subset or the background subset) with intensities between the above mentioned upper and lower intensity thresholds, and to adjust the intensities of the pixels in the intermediate subset to a value range between the maximum and minimum levels as mentioned above.
In the present example, the foreground, background, and intermediate pixel subsets are identified and adjusted by applying a sigma-shaped curve intensity adjustment function to the image 400. Regarding FIG. 5A, an exemplary sigma-shaped function curve is illustrated. When the pre-processor adjusts 200 pixel intensities
BE2018 / 5620 based on such a curve, the curve maps the original intensity (horizontal axis in FIG. 5A) of each pixel to an adjusted intensity (vertical axis in FIG. 5A). As will become apparent, the pixel intensities are shown as values between 0 and 1, rather than between 0 and 255. For the exemplary function of FIG. 5A, such a format results in each pixel having an original intensity greater than about 0.3 being adjusted to an intensity of 1 (i.e., white). As will be understood by a person skilled in the art, the original and adjusted intensities may deviate from the examples given above depending on the data format used by the image 400. Pixels with original intensities above about 0.3 are the foreground subset. Application of the function shown in FIG. 5A also results in that each pixel with an intensity lower than about 0.06 is adjusted to an intensity of 0 (i.e., black). Pixels with original intensities below approximately 0.06 are the background subset. Finally, pixels with original intensities between about 0.06 and about 0.3 become members of the intermediate subset, and are adjusted according to the curve shown in FIG. 5A.
FIG. 5B shows an adjusted image 500 generated by applying the adjustment function of FIG. 5A on the image 400. As can be seen in FIG. 5B, the distribution of intensities in the image 500 becomes more heavily weighted toward extreme extremes (i.e., toward white and black) than in the image 400. The above-mentioned adjustments based on the upper and lower intensity thresholds are based on an assumption that ambient light (i.e. that is, light from environmental sources other than the device 103) is either absent, or is of a significantly lower intensity (e.g., with two or more orders of magnitude) than illumination of the shelves 110 provided by the device 103. The above assumption, combined with the knowledge that the device 103 moves through aisles of shelves 110 to obtain the image 400 and other images, indicates that the objects are closest to the
BE2018 / 5620 device 103 at the time of image acquisition when the lightest objects will appear in the obtained image. Furthermore, it is believed that the shelf edges 118, being the furthest end of each shelf toward the aisle, are the closest (or one of the closest) objects to the device 103 during image acquisition. It follows that the shelf edges 118 are expected to be one of the most light objects (i.e., with the highest intensities) shown in the obtained images. Therefore, the contrast adjustment performed at block 310 emphasizes the lightest objects in the image 400 (which is believed to belong to the shelf edges 118) by further highlighting them, while the darkest objects in the image 400 (which is believed to be the shelf edges 118) not belong) are de-emphasized by further obscuring them.
In some examples, the parameters defining the sigma-shaped curve may be predetermined within the pre-processor 200. In other examples, one or more of the parameters defining the sigma-shaped curve may be based on the image 400 itself. Now looking at FIG. 6, a method 600 of executing block 315 is shown, in which the pre-processor 200 is arranged to update dynamic contrast adjustment parameters.
At block 605, the pre-processor 200 is arranged to compress to generate a compressed version of the image 400. As will become apparent in the discussions below, the image 400 is not replaced by the compressed version, since the compressed version is only used to set contrast adjustment parameters. In the present example, compression includes generating a single horizontal line of pixels, wherein each pixel has an intensity corresponding to the median of an entire column of pixels of the image 400. At block 610, the pre-processor 200 is arranged to identify foreground and background pixel subsets in it
BE2018 / 5620 compressed image by comparing compressed pixel intensities with standard highest and lower intensity thresholds. The standard threshold values can be determined by, for example, standard parameters defining the sigma-shaped curve adjustment function.
At block 615, the pre-processor 200 is arranged to determine whether respective populations of the foreground and background subsets satisfy target populations. For example, the pre-processor can store 200 target populations, expressed as a part of the total number of pixels in the compressed image, for one or both of the foreground and background subsets. At block 615, the pre-processor 200 is therefore arranged to determine an actual population (as a part of the total number of pixels in the compressed image) of pixels that satisfy the standard upper intensity threshold and / or the standard lower intensity threshold.
If the foreground and background populations do not meet the above goals, the application of method 600 proceeds to block 620, where the adjustment parameters deployed to generate the standard threshold values deployed at block 610 are adjusted. For example, as will be appreciated by those skilled in the art, a sigma-shaped curve is partly defined by a parameter referred to as beta, which defines the horizontal position of the curve shown in FIG. 5A along the "original intensity" axis. At block 620, the pre-processor 200 may hence be arranged to adjust the beta parameter up or down based on the comparison of the actual foreground and / or background populations with the target populations. The pre-processor 200 then returns to block 610 to identify the revised foreground and background subsets based on the updated threshold values.
BE2018 / 5620
When the determination is confirmed at block 615, the pre-processor 615 proceeds to block 625, where the current adjustment parameters (i.e., those applied to the most recent execution of block 610) are applied to the entire image 400. After adjustment of the image 400 the preprocessor 200 returns to method 300, and specifically to block 315. The process of updating adjustment parameters from above can also be applied to other sigma-shaped parameters, such as an alpha parameter which is a slope of the curve shown in FIG. 5A. Furthermore, the process shown in FIG. 6 can also be applied to adjustment functions other than a sigma-shaped curve.
Returning to FIG. 3, the selector 204 of the control application 128 is arranged to determine gradients for each of the pixels in the contrast-adjusted image 500. In particular, the selector 204 is adapted to determine at least one vertical gradient for each pixel in the adjusted image. 500. In the current example, the vertical gradients are determined by applying a Sobel filter to the adjusted image 500. The Sobel filter determines both vertical and horizontal gradients, and the horizontal gradients are discarded. Other suitable operations can also be applied to the adjusted image 500 to determine gradients. In general, as will be apparent to those skilled in the art, the gradient for each pixel is determined based on the adjacent pixels (e.g. based on a 3x3 pixel kernel centered around the pixel for which the gradient is determined), and is expressed as a vector or a derivative giving an indication of direction and magnitude of the greatest change in intensity of the pixel under consideration relative to the adjacent pixels.
Vertical gradients are determined at block 315 in the current example because it is assumed that the shelf edges 118 are displayed horizontally in the images captured by the device 103. The
BE2018 / 5620 shelf edges 118 are therefore expected to manifest as vertical gradients in the obtained images because, where pixels on the shelf edges are relatively light, and pixels above or below the shelf edges are relatively dark. For example, referring to FIG. 5B, the pixels in the region 504 will tend to display weak vertical gradients and strong horizontal gradients, while the pixels in the region 508 (which coincides with a shelf edge 118) will tend to display strong vertical gradients and weak horizontal gradients. More generally, therefore, the gradients are determined at block 315 for those which are substantially perpendicular to an expected orientation of the shelf edges 118.
After determining gradients for each pixel in the adjusted image 500, the selector 204 is arranged to select a candidate set of edge pixels based on the gradients. In particular, the selector 204 is arranged to perform a non-maximum suppression operation on the adjusted image 500 to retain only the largest vertical gradient in each of a plurality of pieces of the adjusted image 500. For example, the selector 204 may be arranged to divide the adjusted image 500 into a plurality of 5x5 pixel pieces, and to select only the largest gradient in each piece for inclusion in the candidate set of pixels. The selector 204 can also apply a threshold value at block 315 such that each pixel is not included in the candidate set even if it is a local maximum if the pixel does not meet a minimum vertical gradient threshold. Again referring to the example shown in FIG. 5B, the candidate set of pixels will include at least some of the pixels in the region 508, but may include less (or none) of the pixels in the region 504. The selector 204 is arranged to pass a list of the candidate set of pixels, including pixel positions and vertical gradients, to the line generator 208.
After having received the set of candidate pixels and associated gradient information, the line generator 208 is arranged to be a multiple
BE2018 / 5620 to lay shelf candidate lines over the adjusted image 500. The shelf candidate lines are derived from the candidate pixels. More specifically, each shelf candidate line comprises at least a portion of the candidate set of pixels. In the present example, the line generator 208 is arranged to derive the shelf candidate lines by applying an adjusted Hough transformation to the candidate set of pixels (not the entire adjusted image 500) and their gradients determined at block 315. Typically, a Hough transformation each of a plurality of pixel positions and corresponding gradients (to an identified edge pixel and its gradient can be referred to as a edgel) to one or more of an amount of bins. Each bin corresponds to a set of parameters that define a candidate line within the source image. For example, the parameters generally include a distance ρ which determines the distance from the origin (in the image reference frame) to the point on the candidate line closest to the origin. The parameters also generally include an angle θ which defines the angle between the horizontal axis (i.e. the x-axis) and the line connecting the candidate line with the origin.
FIG. 7A shows a method 700 performed by the line generator 208 in the present example to generate shelf candidate lines from the candidate set of pixels and gradients identified at block 315. At block 705, the line generator 208 is arranged to designate each of the candidate set of pixels, based on its position in the adjusted image 500 and its gradient, on a single or a plurality of bins, each of which defines parameters for a candidate line. That is, no candidate pixel is assigned to more than one bin, unlike a conventional application of a Hough transformation. In other examples, some or all of the adjustments of the Heat Transformation as discussed herein may be omitted. Further discussion of the aforementioned adjustments is provided
BE2018 / 5620
Section 4.3.2 on pages 251-254 of “Computer Vision: Algorithms and Applications”, R. Szeliski, ISBN: 978-1-84882-934-3 (2011), the contents of which are hereby incorporated by reference.
The assignment of candidate gels (i.e., candidate pixels combined with their respective gradients) is performed by identifying, for each edgel, which bin corresponds to a candidate line that includes the edgel or better fits the edgel (if no candidate lines include the edgel). For example, the line generator 208 may be arranged to determine a distance between a given edgel and the origin of the adjusted image 500, as well as an angle between the image x axis and a line extending from the origin to the edgel. The line generator 208 can be arranged to subsequently select the bin with parameters most similar to the distance and angle determined above. The above process is repeated for each of the candidate gels. Each bin is determined by a counter that increases for each edgel assigned to that bin. In this way, densely populated trays indicate that large amounts of pixels match the line parameters corresponding to those trays.
At block 710, the line generator 208 is arranged to select one or more subsets of candidate pixels from the set of candidate pixels. The selected candidate subsets are those assigned to bins with pixel populations that exceed a predetermined threshold value. In the current example, the line generator 208 is arranged to locate local maxima (e.g., a predetermined number of local maxima, such as five) between the baking counts after the application of block 710. The line generator 208 is also arranged to each of the above-mentioned local maxima to discard that do not exceed the predetermined threshold value.
Referring to FIG. 7B, exemplary results of the adjusted Hough transformation are shown in a heat map 720, wherein each
BE2018 / 5620 point represents a bin (that is, a candidate line defined by a distance ρ and an angle θ). The intensity of the point represents the amount of edgels assigned to that bin, as indicated by the scale 724. Five local maxima, also referred to as peaks 728, are indicated. At block 710, the line generator 208 is arranged to select five subsets of candidate pixels corresponding to the pixels assigned to each of the bins identified as peaks 728.
At block 715, the line generator 208 is arranged to generate a shelf line candidate from each selected subset of candidate pixels. As will be apparent to those skilled in the art, this approach is different from that conventionally used with Hough transformations, in which the line parameters ρ and θ are used to generate a line. In the present example, the line generator 208, on the other hand, does not use the parameters ρ and θ, but instead applies an appropriate line fitting operation to each subset of pixels to generate a shelf candidate line including the pixels of the subset. After having generated the shelf candidate lines, the line generator 208 passes on indicators of the lines (e.g., line end pixel positions within the image reference frame) to the region generator 212. The execution of the method 700 ends, and the execution of the method 300 continues at block 325. Briefly to FIG. 8, the adjusted image 500 is shown with five candidate lines 800-1, 800-2, 800-3, 800-4 and 800-5 superimposed, corresponding to the peaks 728 shown in FIG. 7B.
At block 325, the region generator 212 is arranged to identify at least a pair of candidate board lines 800 that satisfy a predetermined sequence of intensity transitions. As will become apparent from FIGS. 4 and 5B, the boundaries of shelf edges 118 appear in obtained images as transitions from dark to light or from light to dark. More
BE2018 / 5620 specifically, a transition from dark to light, followed by an opposite transition, is indicative of a shelf edge 118 between the transitions. The region generator 212 is arranged to identify at least a pair of the candidate boarding lines 800 with the above-mentioned transition order (i.e., dark to light, followed by light to dark).
Turning to FIG. 9, a method 900 for identifying pairs of candidate board lines is shown, as performed by the region generator 212. At block 905, the region generator 212 is arranged to select a candidate board line 800 as a primary line, starting from a pre-configured side of the image 500 and traversing the image 500 in a pre-configured direction. For example, the region generator 212 may be arranged to start at the top of the image 500, in the orientation shown in FIG. 8, and thus to select the line 800-1 at block 905. At block 910, the region generator 212 is arranged to determine whether the selected primary line has an expected first transition. In the current example, the first expected transition is a transition from dark (i.e., low intensity) to light (i.e., high intensity). The nature of the transition can be determined by the region generator 212, for example by referring to the previously determined gradients for the candidate pixels used to define the selected candidate line.
In the present example, the determination at block 910 is positive with respect to line 800-1, and the execution method 900 therefore proceeds to block 915. If the determination at block 910 is negative, the next candidate boarding line is selected at block 905.
At block 915 a secondary line is selected, in the form of the next line (after the current primary line) in the above-mentioned travel direction. In the current example, as seen in FIG. 8, the line 800-2 is the next line after the line 800-1. At block 920, the region generator becomes 212
BE2018 / 5620 arranged to determine whether the selected secondary line has an expected second transition, which is typically opposite to the first transition. Thus, in the current example, the region generator 212 determines at block 915 whether the line 800-2 has a light to dark transition. If the determination at block 920 is negative, the next secondary line is selected (i.e., the region generator 212 returns to block 915). However, in the current exemplary embodiment of the method 900, the determination at block 920 is positive, and the region generator 212 therefore proceeds to block 925.
At block 925, the region generator 212 can also be arranged to determine whether a distance between the primary and secondary lines satisfies one or more threshold values. For example, the region generator 212 may store one or both of a pre-configured highest threshold value, above which a pair of candidate shelving lines are too far away to represent a shelf edge 118, and a pre-configured lower threshold value, among which a pair of candidate shelving lines are too close to each other around a shelf edge 118. If the distance threshold value (s) evaluated at block 925 is not met, a further secondary line is selected at block 915 (or, if no further secondary lines are available, the next primary line is selected at block 905). However, if the distance threshold value is met at block 925, the region generator 212 proceeds to block 930, the region generator 212 being arranged to store identifiers of the current pair of primary and secondary candidate shelf lines, for example, as a list of line pairs likely to be the shelf edges 118. introduce.
The region generator 212 is then arranged to determine at block 935 whether primary lines are left to be processed. Shelf candidate lines that have not yet been selected as primary lines at block 905, or that have not been successfully paired as secondary lines,
BE2018 / 5620 are processed as further primary lines via further versions of blocks 905-930. Referring again to FIG. 8, it will be appreciated that the line 800-3 satisfies the determination at block 910, but that none of the lines 800-4 and 800-5 successfully pair with the line 800-3 (the line 800-4 has the wrong transition, and the line 800-5 is too far away from the line 800-3). However, lines 800-4 and 800-5 are identified as another pair and stored at block 930.
Returning to FIG. 3, at block 330, the region generator 212 is arranged to generate and store boundary frames corresponding to the pairs of lines identified at block 325. The boundary boxes generated at block 330 identify the locations of the shelf edges 118 in the image data. In some examples, the region generator 212 is arranged to generate a boundary frame for each pair identified by the end points of the lines in the pair (i.e., four end points defining a boundary frame). In other examples, however, the region generator 212 may be arranged to further fine tune the region of the adjusted image 500, for example via a method 1000 shown in FIG. 10.
At block 1005, the region generator 212 is arranged to extend each line pair to the edges of the adjusted image 500. Referring to FIG. 11, the adjusted image 500 is shown with extended lines 1100-1, 1100-2, 1100-4, and 1100-5, corresponding to lines 800-1, 8002, 800-4, and 800-5. The initial boundary frame is the area between the image edges and the extended lines 1100.
At block 1010, the region generator 212 is arranged to perform a morphological opening operation on the surface of the adjusted image 500 within everyone's initial boundary frame. The morphological opening operation may include, for example, the application of a threshold value (to convert the initial boundary frame to binary intensities) followed by erosion and dilation.
BE2018 / 5620
FIG. 11 shows the results of block 1010. In particular, a shelf mask 1104-1, 1104-2 is generated via the morphological opening operation.
At block 1015, the region generator 212 is arranged to determine a refined boundary frame by fitting a frame to the aforementioned shelf mask.
Looking at FIG. 12, refined boundary frames 1200-1 and 1200-2 are displayed over the adjusted image 500. When the refining boundary frames 1200 is completed, the region generator 212 is arranged to store the boundary frames (e.g., by storing the image coordinates of the corners of each frame) in the memory 122 (for example, in the storage location 132).
Specific embodiments have been described in the foregoing description. However, it will be apparent to one skilled in the art that multiple modifications and changes can be made without departing from the scope of the inventions set forth in the claims below. Accordingly, the description and the figures must be viewed in a clarifying, rather than limiting, sense, and all such modifications are intended to be included in the scope of the present teachings. For clarity and concise description, features are described herein as part of the same or separate embodiments, however, it will be understood that the scope of the invention may include embodiments with combinations of all or some of the features described. It may be understood that the embodiments shown have the same or identical parts, except where they are described as being different.
The benefits, benefits, solutions to problems, and any (/ all) element (s) that (can) cause any benefit, benefit, or solution or can become more pronounced are not to to
BE2018 / 5620 are explained as a crucial, required, or essential feature or component in any one or all of the claims. The invention is only defined by the appended claims, including some improvements made during the granting procedure of this application and all equivalents of those claims as issued.
In addition, in this document, related terms such as first and second, above and below, and the like, may only be used to distinguish an entity or action from another entity or action, without necessarily requiring or implying an actual relationship or order between such entities or promotions. The terms "include," "including," "has," "including," "includes," "includes," "includes," or any other variation thereof, are intended as a non-exclusive inclusion, so that a process, method, article, or device that includes, includes, contains a list of elements, not only contains those elements, but possibly contains other elements that are not explicitly listed or that are not inherent in such a process, method, article, or device. An element preceded by "includes ... a," "has ... a," "includes ... a," "contains ... a," without limitation, does not exclude the existence of additional identical elements in the process, method, article, or device that has the element, is provided with, or contains. The term "one" is defined as one or more unless explicitly stated otherwise herein. The terms "substantially," "essential," "approximately," or any other variation thereof, are defined as being close to that understood by a person skilled in the art, and in a certain non-limiting embodiment, the term is defined as being within 10%, in another embodiment as being within 5%, in another embodiment as being within 1%, and in another embodiment as being within 0.5%. The term "coupled" as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that / that
BE2018 / 5620 is “configured” in a certain way, at least configured that way, but may also be configured in ways that are not listed.
It will be appreciated that some embodiments may include one or more generic or specialized processors (or "processing devices") such as microprocessors, digital signal processors, custom processors, and FPGAs (field-programmable gate arrays), and unique, stored program instructions (including both software and firmware) that control one or more processors to perform, in combination with certain circuitry without processors, some, most, or all functions of the method and / or device described herein. Alternatively, some or all of the functions may be performed by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs), in which each function or some combinations of the functions are implemented as custom logic. Naturally, a combination of the two approaches can be used.
In addition, an embodiment may be embodied as a computer-readable storage medium provided with computer-readable code stored thereon for programming a computer (e.g., including a processor) to perform a method as described and claimed herein. Examples of such computer program products include, but are not limited to, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (read-only memory), a PROM (programmable read-only memory), an EPROM (erasable programmable) read-only memory), an EEPROM (electrically erasable programmable read-only memory) and a flash memory. Furthermore, it is to be expected that a skilled person, notwithstanding possible significant difficulties and many design choices, will be motivated by, for example, available time, current
BE2018 / 5620 technology, and economic considerations, guided by the concepts and principles described herein, with minimal experimentation, will easily be able to generate such software instructions and programs and integrated circuits.
The extract of the description is provided to allow the reader to quickly ascertain the nature of the technical description. It is submitted with the intention that it will not be used to explain or limit the scope of protection or the meaning of the claims. In addition, it can be seen in the foregoing detailed description that various features are grouped together in various embodiments for the purpose of simplifying the description. This method of describing should not be interpreted as expressing the intention that the claimed embodiments require more features than are explicitly stated in each claim. Rather, as the following claims show, inventive subject matter is contained in less than all the features of a single described embodiment. Thus, the following claims are hereby incorporated in the detailed description, with each claim standing alone as a separately claimed subject matter. The mere fact that certain measures are cited in different claims gives no indication that a combination of these measures cannot be used to achieve an advantage. Many variants will be clear to the skilled person. All variants must be construed as being included in the scope of the invention as defined in the following claims.

权利要求:
Claims (19)
[1]
A method for detecting a shelf edge by an image processing controller, comprising:
obtaining, by the image processing control, an image representing a shelf;
increasing an intensity of a foreground subset of image pixels in the image, wherein each pixel in the foreground subset of pixels has an initial intensity that exceeds an upper intensity threshold, and decreasing an intensity of a background subset of pixels in the image, wherein each pixel in the image background subset of pixels has an initial intensity that is lower than a lower intensity threshold;
in response to increasing and decreasing, (i) determining gradients for each of the pixels and (ii) selecting a candidate set of the pixels based on the gradients;
superimpose a plurality of shelf candidate lines over the image derived from the candidate set of pixels;
identify a pair of shelf candidate lines that satisfy a predetermined sequence of intensity transitions; and generating a shelf edge boundary frame corresponding to the pair of shelf candidate lines.
[2]
The method of claim 1, further comprising downsampling the image prior to increasing and decreasing.
[3]
The method of claim 1 or 2, further comprising: adjusting an intensity of an intermediate subset of pixels with intensities between the upper intensity threshold and the lower intensity threshold.
[4]
The method of claim 3, further comprising:
BE2018 / 5620 prior to increasing, decreasing and adjusting, determining intensity adjustment parameters by comparing pixel populations of the foreground subset and the background subset with foreground and background target populations.
[5]
5 distance criterion.
The computing device of claim 18, wherein the distance criterion defines a minimum distance between the pair of shelf candidate lines and a maximum distance between the pair of shelf candidate lines.
The method of any one of the preceding claims, wherein selecting the candidate set of pixels comprises selecting pixels which locally represent maximum gradients in an orientation perpendicular to an expected orientation of the shelf edge.
[6]
The method of any one of the preceding claims, wherein overlapping the plurality of shelf candidate lines comprises: assigning each pixel in the candidate set to one of a plurality of bins, each bin corresponding to a line;
selecting candidate subsets of the pixels assigned to respective bins with pixel populations that exceed a population threshold value; and generating a shelf candidate line from each candidate subset of pixels.
[7]
The method of any one of the preceding claims, wherein the identifying further comprises:
traversing the image in a predetermined direction; and evaluating successive pairs of the shelf candidate lines against the intensity sequence.
[8]
The method of any one of the preceding claims, wherein the identifying further comprises determining whether the pair of shelf candidate lines that meets the predetermined sequence meets a distance criterion.
[9]
The method of claim 8, wherein the distance criterion defines a minimum distance between the pair of shelf candidate lines and a maximum distance between the pair of shelf candidate lines.
[10]
20. The computer device of any one of the preceding claims 11-
The method of any one of the preceding claims, further comprising storing the shelf edge bounding frame in a memory.
BE2018 / 5620
[11]
A computer device for detecting shelf edges, comprising:
a memory; and an image processing control coupled to the memory, the image processing control comprising:
a pre-processor arranged to obtain an image representing a shelf, and to increase an intensity of a foreground subset of pixels in the image, wherein each pixel in the foreground subset of pixels has an initial intensity that exceeds an upper intensity threshold, and lowers an intensity of a background subset of pixels in the image, wherein each pixel in the background subset of pixels has an initial intensity that is lower than a lower intensity threshold;
a selector arranged to, in response to the raising and lowering, (i) determine gradients for each of the pixels and (ii) select a candidate set of the pixels based on the gradients;
a line generator adapted to superimpose a plurality of shelf candidate lines over the image derived from the candidate set of pixels; and a region generator adapted to identify a pair of the shelf candidate lines that satisfy a predetermined sequence of intensity transitions; and the image processing controller adapted to generate a shelf edge boundary frame corresponding to the pair of shelf candidate lines in the memory.
[12]
The computer apparatus of claim 11, wherein the pre-processor is further adapted to downsample the image prior to raising and lowering.
[13]
The computer device of claim 11 or 12, wherein the pre-processor is further arranged to:
BE2018 / 5620 to adjust an intensity of an intermediate subset of pixels with intensities between the upper intensity threshold and the lower intensity threshold.
[14]
14, wherein the selector is further adapted to select the candidate set of pixels by selecting pixels which locally display maximum gradients in an orientation perpendicular to an expected orientation of the shelf edge.
The computer device of claim 13, wherein the processor is further adapted to:
prior to increasing, decreasing, and adjusting, determine intensity adjustment parameters by comparing pixel populations of the foreground subset and the background subset with foreground and background target populations.
[15]
15, wherein the line generator is arranged to overlap the plurality of shelf candidate lines by:
assigning each pixel in the candidate set to one of a plurality of trays each corresponding to a line;
selecting candidate subsets of the pixels assigned to respective bins with pixel populations that exceed a threshold value; and generating a shelf candidate line from each candidate subset of pixels.
The computer device of any of the preceding claims 11-
[16]
The computer device of any of the preceding claims 11-
[17]
17, wherein the region generator is further adapted to identify the pair of shelf candidate lines by: determining whether the pair of shelf candidate lines that meets the predetermined sequence meets a
The computer apparatus of any of the preceding claims 1116, wherein the region generator is adapted to identify the pair of shelf candidate lines by:
traversing the image in a predetermined direction; and evaluating successive pairs of the shelf candidate lines against the intensity sequence.
BE2018 / 5620
[18]
The computer device of any of the preceding claims 11-
[19]
19, wherein the image processing controller is further arranged to store the shelf edge bounding frame in the memory.

类似技术:

---

公开号 | 公开日 | 专利标题

---

BE1025892A9|2019-08-07|METHOD FOR SCREEN EDGE DETECTION

---

US9961256B2|2018-05-01|Apparatus and method for specifying and aiming cameras at shelves

---

US10726273B2|2020-07-28|Method and apparatus for shelf feature and object placement detection from shelf images

---

BE1026149B1|2020-07-03|METHOD, SYSTEM AND APPARATUS FOR CORRECTING TRANSPARENCY ARTEFACTS IN DATA REPRESENTING A SUPPORT STRUCTURE

---

JP6222345B2|2017-11-01|POS terminal device, POS system, image processing method and program

---

CN108885259A|2018-11-23|Use the method and system of line scanner processing point cloud data

---

BE1026263A9|2020-01-07|SYSTEM AND METHOD FOR DETERMINING A LOCATION FOR PLACING A PACKAGE

---

BE1026160A1|2019-10-22|METHOD AND CALCULATOR FOR OBTAINING LABEL POSITIONS

---

BE1026161A9|2019-10-29|METHOD, SYSTEM AND DEVICE FOR MOBILE AUTOMATION DEVICE LOCALIZATION

---

EP3563345B1|2020-11-18|Automatic detection, counting, and measurement of lumber boards using a handheld device

---

US20210272316A1|2021-09-02|Method, System and Apparatus for Object Detection in Point Clouds

---

BE1027283B1|2021-02-25|METHOD, SYSTEM AND DEVICE FOR DETECTING PRODUCT VIEWS

---

AU2019396253A1|2021-04-29|Method, system and apparatus for auxiliary label detection and association

---

JP2021128798A|2021-09-02|Information processing device, system, image processing method and program

---

JP2022512604A|2022-02-07|Methods, systems and equipment for determining the depth of the support structure

---

US20210183095A1|2021-06-17|Method, System and Apparatus for Detecting Item Facings

---

WO2020247067A1|2020-12-10|Method, system and apparatus for shelf edge detection

---

WO2020210822A1|2020-10-15|System and method for associating products and product labels

---

CN112560869A|2021-03-26|Model obtaining method, device and equipment

---

CN112560616A|2021-03-26|Image recognition method, device and equipment

同族专利:

---

公开号 | 公开日

---

GB2567922A|2019-05-01|

---

GB2567922B|2020-12-02|

---

BE1025892A9|2019-08-07|

---

US20200082202A1|2020-03-12|

---

US20190073559A1|2019-03-07|

---

BE1025892A1|2019-08-01|

---

US10489677B2|2019-11-26|

---

GB201813580D0|2018-10-03|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

US20020097439A1|2001-01-23|2002-07-25|Oak Technology, Inc.|Edge detection and sharpening process for an image|

---

US6026376A|1997-04-15|2000-02-15|Kenney; John A.|Interactive electronic shopping system and method|

---

US6975764B1|1997-11-26|2005-12-13|Cognex Technology And Investment Corporation|Fast high-accuracy multi-dimensional pattern inspection|

---

US6711293B1|1999-03-08|2004-03-23|The University Of British Columbia|Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image|

---

WO2002005121A2|2000-07-11|2002-01-17|Mediaflow, Llc|System and method for calculating an optimum display size for a visual object|

---

US7233699B2|2002-03-18|2007-06-19|National Instruments Corporation|Pattern matching using multiple techniques|

---

US7168618B2|2004-08-12|2007-01-30|International Business Machines Corporation|Retail store method and system|

---

US20080175513A1|2005-04-19|2008-07-24|Ming-Jun Lai|Image Edge Detection Systems and Methods|

---

JP4730121B2|2006-02-07|2011-07-20|ソニー株式会社|Image processing apparatus and method, recording medium, and program|

---

AT470912T|2006-04-28|2010-06-15|Toyota Motor Europ Nv|ROBUST DETECTOR AND DESCRIPTOR FOR ONE POINT OF INTEREST|

---

US7693757B2|2006-09-21|2010-04-06|International Business Machines Corporation|System and method for performing inventory using a mobile inventory robot|

---

US7982423B2|2007-07-04|2011-07-19|Bossa Nova Concepts, Llc|Statically stable biped robotic mechanism and method of actuating|

---

US8189855B2|2007-08-31|2012-05-29|Accenture Global Services Limited|Planogram extraction based on image processing|

---

US9135491B2|2007-08-31|2015-09-15|Accenture Global Services Limited|Digital point-of-sale analyzer|

---

US8630924B2|2007-08-31|2014-01-14|Accenture Global Services Limited|Detection of stock out conditions based on image processing|

---

US8091782B2|2007-11-08|2012-01-10|International Business Machines Corporation|Using cameras to monitor actual inventory|

---

US7971664B2|2008-03-18|2011-07-05|Bossa Nova Robotics Ip, Inc.|Efficient actuation and selective engaging and locking clutch mechanisms for reconfiguration and multiple-behavior locomotion of an at least two-appendage robot|

---

US8064729B2|2008-04-03|2011-11-22|Seiko Epson Corporation|Image skew detection apparatus and methods|

---

US7997430B2|2009-06-30|2011-08-16|Target Brands, Inc.|Display apparatus and method|

---

CA2754159C|2009-08-11|2012-05-15|Certusview Technologies, Llc|Systems and methods for complex event processing of vehicle-related information|

---

JP4914528B1|2010-08-31|2012-04-11|新日鉄ソリューションズ株式会社|Augmented reality providing system, information processing terminal, information processing apparatus, augmented reality providing method, information processing method, and program|

---

US9171442B2|2010-11-19|2015-10-27|Tyco Fire & Security Gmbh|Item identification using video recognition to supplement bar code or RFID information|

---

CN103443802B|2011-01-24|2016-12-14|数据逻辑Adc公司|For reading the system and method for optical code|

---

US20120190453A1|2011-01-25|2012-07-26|Bossa Nova Robotics Ip, Inc.|System and method for online-offline interactive experience|

---

US20120191880A1|2011-01-26|2012-07-26|Bossa Nova Robotics IP, Inc|System and method for identifying accessories connected to apparatus|

---

JPWO2012132324A1|2011-03-31|2014-07-24|日本電気株式会社|Store system, control method thereof, control program, and information access system|

---

US9785898B2|2011-06-20|2017-10-10|Hi-Tech Solutions Ltd.|System and method for identifying retail products and determining retail product arrangements|

---

US9129277B2|2011-08-30|2015-09-08|Digimarc Corporation|Methods and arrangements for identifying objects|

---

TWI622540B|2011-09-09|2018-05-01|辛波提克有限責任公司|Automated storage and retrieval system|

---

US8706293B2|2011-11-29|2014-04-22|Cereson Co., Ltd.|Vending machine with automated detection of product position|

---

AU2013207414B2|2012-01-06|2016-02-11|Sunrise R&D Holdings, Llc|Display shelf modules with projectors for displaying product information and modular shelving systems comprising the same|

---

US9418352B2|2012-06-29|2016-08-16|Intel Corporation|Image-augmented inventory management and wayfinding|

---

US20140003655A1|2012-06-29|2014-01-02|Praveen Gopalakrishnan|Method, apparatus and system for providing image data to represent inventory|

---

US8971637B1|2012-07-16|2015-03-03|Matrox Electronic Systems Ltd.|Method and system for identifying an edge in an image|

---

WO2014066422A2|2012-10-22|2014-05-01|Bossa Nova Robotics Ip, Inc.|Self-deploying support member, and methods and apparatus using same|

---

US20140195373A1|2013-01-10|2014-07-10|International Business Machines Corporation|Systems and methods for managing inventory in a shopping store|

---

US20140214547A1|2013-01-25|2014-07-31|R4 Technologies, Llc|Systems and methods for augmented retail reality|

---

WO2014181323A1|2013-05-05|2014-11-13|Trax Technology Solutions Pte Ltd.|System and method of retail image analysis|

---

US9158988B2|2013-06-12|2015-10-13|Symbol Technclogies, LLC|Method for detecting a plurality of instances of an object|

---

US9886678B2|2013-09-25|2018-02-06|Sap Se|Graphic representations of planograms|

---

US20150262116A1|2014-03-16|2015-09-17|International Business Machines Corporation|Machine vision technology for shelf inventory management|

---

EP3152010A4|2014-06-04|2018-01-24|Intelligrated Headquarters LLC|Truck unloader visualization|

---

US9659204B2|2014-06-13|2017-05-23|Conduent Business Services, Llc|Image processing methods and systems for barcode and/or product label recognition|

---

US9928708B2|2014-12-12|2018-03-27|Hawxeye, Inc.|Real-time video analysis for security surveillance|

---

EP3437031A4|2016-03-29|2019-11-27|Bossa Nova Robotics IP, Inc.|SYSTEM AND METHOD FOR LOCATING, IDENTIFYING AND COUNTING lTEMS|

---

CN109328359A|2016-06-30|2019-02-12|波萨诺瓦机器人知识产权有限公司|Multi-camera system for inventory tracking|

---

US10785418B2|2016-07-12|2020-09-22|Bossa Nova Robotics Ip, Inc.|Glare reduction method and system|

---

US20180053091A1|2016-08-17|2018-02-22|Hawxeye, Inc.|System and method for model compression of neural networks for use in embedded platforms|

---

US20180101813A1|2016-10-12|2018-04-12|Bossa Nova Robotics Ip, Inc.|Method and System for Product Data Review|

---

US20180114183A1|2016-10-25|2018-04-26|Wal-Mart Stores, Inc.|Stock Level Determination|

---

CN107067382A|2017-05-11|2017-08-18|南宁市正祥科技有限公司|A kind of improved method for detecting image edge|

---

US20190034864A1|2017-07-25|2019-01-31|Bossa Nova Robotics Ip, Inc.|Data Reduction in a Bar Code Reading Robot Shelf Monitoring System|

---

US10861302B2|2017-08-17|2020-12-08|Bossa Nova Robotics Ip, Inc.|Robust motion filtering for real-time video surveillance|

---

US10783379B2|2017-08-23|2020-09-22|Bossa Nova Robotics Ip, Inc.|Method for new package detection|

---

US20190180150A1|2017-12-13|2019-06-13|Bossa Nova Robotics Ip, Inc.|Color Haar Classifier for Retail Shelf Label Detection|US11042161B2|2016-11-16|2021-06-22|Symbol Technologies, Llc|Navigation control method and apparatus in a mobile automation system|

---

US11055662B2|2017-03-15|2021-07-06|Walmart Apollo, Llc|System and method for perpetual inventory management|

---

US10997552B2|2017-03-15|2021-05-04|Walmart Apollo, Llc|System and method for determination and management of root cause for inventory problems|

---

WO2018204342A1|2017-05-01|2018-11-08|Symbol Technologies, Llc|Product status detection system|

---

US10949798B2|2017-05-01|2021-03-16|Symbol Technologies, Llc|Multimodal localization and mapping for a mobile automation apparatus|

---

US11010920B2|2018-10-05|2021-05-18|Zebra Technologies Corporation|Method, system and apparatus for object detection in point clouds|

---

US11090811B2|2018-11-13|2021-08-17|Zebra Technologies Corporation|Method and apparatus for labeling of support structures|

---

US11003188B2|2018-11-13|2021-05-11|Zebra Technologies Corporation|Method, system and apparatus for obstacle handling in navigational path generation|

---

US11079240B2|2018-12-07|2021-08-03|Zebra Technologies Corporation|Method, system and apparatus for adaptive particle filter localization|

---

US11015938B2|2018-12-12|2021-05-25|Zebra Technologies Corporation|Method, system and apparatus for navigational assistance|

---

US11080566B2|2019-06-03|2021-08-03|Zebra Technologies Corporation|Method, system and apparatus for gap detection in support structures with peg regions|

---

US11200677B2|2019-06-03|2021-12-14|Zebra Technologies Corporation|Method, system and apparatus for shelf edge detection|

---

US20200380454A1|2019-06-03|2020-12-03|Zebra Technologies Corporation|Method, System and Apparatus for Detecting Product Facings|

---

US11151743B2|2019-06-03|2021-10-19|Zebra Technologies Corporation|Method, system and apparatus for end of aisle detection|

---

US11107238B2|2019-12-13|2021-08-31|Zebra Technologies Corporation|Method, system and apparatus for detecting item facings|

---

CN111931723B|2020-09-23|2021-01-05|北京易真学思教育科技有限公司|Target detection and image recognition method and device, and computer readable medium|

---

CN111968145B|2020-10-23|2021-01-15|腾讯科技（深圳）有限公司|Box type structure identification method and device, electronic equipment and storage medium|

法律状态:
2019-10-02| FG| Patent granted|Effective date: 20190912 |

优先权:

---

申请号 | 申请日 | 专利标题

---

US15/698,340|US10489677B2|2017-09-07|2017-09-07|Method and apparatus for shelf edge detection|

---

US15698340|2017-09-07|

---